fix(agent): recover from unsupported image input instead of poisoning the turn - #4896
Conversation
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
|
|
||
| fn is_unsupported_image_input_error(body: &str) -> bool { | ||
| body.to_ascii_lowercase() | ||
| .contains("no endpoints found that support image input") |
There was a problem hiding this comment.
This seems too fragile?
Maybe break it into a few different errors
contains("not support image") // for open ai models
|| contains("no endpoints found that support image input") // for deepseek models
|| contains("image inputs are not supported") // for claude models
There was a problem hiding this comment.
I think ollama outputs: "not support multi-modal inputs"
There was a problem hiding this comment.
Good instinct — the matcher is deliberately narrow, but we probed the suggestion before taking it, and the phrase list turns out not to be the binding constraint: the status gate is. is_unsupported_image_input_error only runs inside the status == 404 arms (both here and in openrouter_post). We measured all four cases against the real post path:
| phrase | status | result |
|---|---|---|
| OpenAI-style "not support image" | 400 | AgentError::Llm — never reaches the matcher |
| Claude-style "image inputs are not supported" | 400 | AgentError::Llm — never reaches the matcher |
| Claude-style phrase | 404 | LlmModelNotFound — matcher ran, phrase absent |
| DeepSeek/OpenRouter phrase | 404 | UnsupportedImageInput — works today |
So adding the OpenAI/Claude phrases as written would close the comment without changing behavior for either provider — their rejections arrive as 400s and are swallowed upstream of the matcher. Real coverage means hoisting the check above the status dispatch, which is a structurally different patch.
Two more reasons we're holding to the narrow matcher in this PR:
- We can't verify the other phrases. The DeepSeek phrase is in this PR because a live trial produced it verbatim; nobody in this effort has a captured 400 body from OpenAI or Anthropic rejecting an image. Matching a guessed phrase fails silently the moment the real wording differs.
- A false positive here is not benign.
contains("not support image")is broad, and misclassification strips images out of history on a turn where images were fine. The neighboring testopenrouter_post_404_unknown_model_stays_model_not_foundexists precisely because 404-classification mistakes send users to the wrong fix.
Filed as a follow-up: broaden coverage (hoist above the status dispatch + per-provider phrases) once we have real captured rejection bodies to match against. The PR body now states the scope guarantee explicitly.
There was a problem hiding this comment.
Following up on the measurement above — you were right about Ollama, and I have the captured body now.
Ollama is reachable through our openai-compat path with no credential, so I pulled a text-only model and drove the real buzz-agent binary against it at this PR's head. The rejection is real, close to your guess but not identical:
HTTP 400
{"error":{"message":"{\"error\":{\"code\":400,\"message\":\"Multimodal data provided, but model does not support multimodal requests.\",\"type\":\"invalid_request_error\"}}","type":"invalid_request_error","param":null,"code":null}}
You guessed "not support multi-modal inputs"; the actual wording is does not support multimodal requests — no hyphen, "requests" not "inputs". That gap is exactly why I wanted a captured body rather than a guessed phrase: a matcher built on the guess would have silently missed this. Also worth flagging for whoever implements it — the body is doubly encoded, so the real sentence sits in a JSON string nested inside error.message.
At this PR's head the failure is unchanged for Ollama — three consecutive turns die on the same 400, history stays poisoned. I also ran a proxy that rewrote only that 400 into the OpenRouter 404 + phrase: recovery fires and all three turns end end_turn. Same binary, same rig, sole variable is the status and phrase. So the recovery machinery here is right; it just isn't reached.
Two more measurements that constrain the fix, and they argue against the simple version:
- Ollama returns 404 for
model not found(text-only request, no image). - Ollama returns 404 for
model not foundeven when the request does carry an image.
So we can't just add the phrase to the existing 404 arm or loosen that arm — on this provider 404 already means something else. Real coverage needs the check hoisted above the status dispatch, plus a test pinning that Ollama's model-not-found 404 keeps LlmModelNotFound.
Full evidence, controls, and a reproduction recipe are on #4899. OpenAI and Anthropic bodies are still uncaptured, so I'd keep those phrases out until someone has them verbatim.
There was a problem hiding this comment.
Update: your instinct on the phrase list was better than my first reply gave it credit for. We stood up a credential-free live rig against Ollama and captured a real image rejection — and one of your guessed phrases is essentially real:
HTTP 400
{"error":{"message":"{\"error\":{\"code\":400,\"message\":\"Multimodal data provided, but model does not support multimodal requests.\",\"type\":\"invalid_request_error\"}}", ...}}
Two things the capture pins down:
- It arrives as a 400, which confirms the structural point from my earlier reply — adding phrases to the current 404-gated matcher would not have caught it. The check has to hoist above the status dispatch. And Ollama's own 404 (
model 'x' not found) means model-not-found even when the request carries an image, so the 404 classifications must stay as-is. - The body is doubly-encoded (the sentence is a JSON string nested inside
error.message), which any structured matcher needs to know about.
The live rig also confirmed the recovery machinery in this PR works end-to-end against a real provider (a proxy rewriting that 400 to the OpenRouter-shaped 404 produces clean end_turn recovery on the same binary).
Sequencing: this PR lands as-is (correct for OpenRouter/DeepSeek, live-verified recovery), and the hoist + captured Ollama phrase + ordering tests come as a stacked PR tracked in #4899. OpenAI/Anthropic phrases stay out until someone captures real bodies from them.
atishpatel
left a comment
atishpatel
left a comment
There was a problem hiding this comment.
LGTM other than comment above
The `removed == 0` branch in the recovery path is the only thing stopping a phrase-matched 404 with no image in history from re-requesting forever: `max_rounds` defaults to 0 (unlimited) in production, and buzz-acp's queue retry never fires because the turn never returns. Deleting that guard left the whole package suite green, so the branch had no coverage. This test drives the typed error with an empty history and asserts the turn fails with the typed error after exactly one request. Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz> Signed-off-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz>
…er-snapshots * origin/main: fix(workflow): bind trigger author to the signed event (#4607) fix(git): revoke access for banned relay members (#4608) fix(agent): recover from unsupported image input instead of poisoning the turn (#4896) Define private managed agent wire protocol (#4593) fix(mobile): serialize channel sections sync (#3165) fix(desktop): make missing-command error actionable for released builds (#4802) chore(release): release Buzz Desktop version 0.5.5 (#4809) Co-authored-by: Taylor Ho <taylorkmho@gmail.com> Signed-off-by: Taylor Ho <taylorkmho@gmail.com>
Pull main in before remediation, per Tyler's instruction: no rebase, no force, new commits on top. Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz> Signed-off-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz> * origin/main: fix(desktop): remove join API token control (#4897) fix(desktop): allow shared agent mentions (#4913) Polish mobile top navigation (#4778) fix(release): tag immutable desktop candidates (#4811) fix(channels): restrict private-channel invitations (#4612) fix(acp): reject unattended permission requests (#4609) fix(workflow): bind trigger author to the signed event (#4607) fix(git): revoke access for banned relay members (#4608) fix(agent): recover from unsupported image input instead of poisoning the turn (#4896) Define private managed agent wire protocol (#4593) fix(mobile): serialize channel sections sync (#3165) fix(desktop): make missing-command error actionable for released builds (#4802) Signed-off-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz>
…-overflow-recovery * origin/main: fix(buzz-agent): scope handoff cap per turn, not per session lifetime (#4805) Fix mobile message timeline bounce (#4862) Polish mobile bottom sheets and profile cards (#4911) Fix media attachment actions (#4849) fix(desktop): remove join API token control (#4897) fix(desktop): allow shared agent mentions (#4913) Polish mobile top navigation (#4778) fix(release): tag immutable desktop candidates (#4811) fix(channels): restrict private-channel invitations (#4612) fix(acp): reject unattended permission requests (#4609) fix(workflow): bind trigger author to the signed event (#4607) fix(git): revoke access for banned relay members (#4608) fix(agent): recover from unsupported image input instead of poisoning the turn (#4896) Define private managed agent wire protocol (#4593) fix(mobile): serialize channel sections sync (#3165) fix(desktop): make missing-command error actionable for released builds (#4802) chore(release): release Buzz Desktop version 0.5.5 (#4809) feat: paste composer text without formatting (#4801) Revert "chore(release): release Buzz Desktop version 0.5.5" (#4808) chore(release): release Buzz Desktop version 0.5.5 (#4800) Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> # Conflicts: # crates/buzz-agent/src/agent.rs # crates/buzz-agent/src/handoff.rs # crates/buzz-agent/src/types.rs # crates/buzz-agent/tests/regressions.rs
…arer-auth * origin/main: (65 commits) fix(desktop): route macos notification clicks (#4799) feat(mobile): sync themes per community (#3767) feat(desktop): sync themes per community (#3653) feat(desktop): cap OpenClaw agent parallelism at 5 (#4019) fix(buzz-agent): scope handoff cap per turn, not per session lifetime (#4805) Fix mobile message timeline bounce (#4862) Polish mobile bottom sheets and profile cards (#4911) Fix media attachment actions (#4849) fix(desktop): remove join API token control (#4897) fix(desktop): allow shared agent mentions (#4913) Polish mobile top navigation (#4778) fix(release): tag immutable desktop candidates (#4811) fix(channels): restrict private-channel invitations (#4612) fix(acp): reject unattended permission requests (#4609) fix(workflow): bind trigger author to the signed event (#4607) fix(git): revoke access for banned relay members (#4608) fix(agent): recover from unsupported image input instead of poisoning the turn (#4896) Define private managed agent wire protocol (#4593) fix(mobile): serialize channel sections sync (#3165) fix(desktop): make missing-command error actionable for released builds (#4802) ... Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> # Conflicts: # CHANGELOG.md
Brings the bench branch (forked from main at 3d7712c, Aug 1) up to current main (f53bbd1) as a merge commit — no rebase, no history rewrite — so the branch stops silently drifting behind agent and runtime fixes. Notably picks up: - #4896: recover from unsupported image input instead of poisoning the turn (the 'Poisoned Polaroid' that tainted 15 tb21 trials) - #5136: mention the orchestrator by pubkey when posting the task (the 'Register Grabber' that killed large-scale-text-editing) One conflict, container_runtime.py: the bench branch's solo-mode _wait_for_done changes vs #5136's _send mention plumbing. Resolved by keeping both (same resolution as the validated cherry-pick 2944a88). Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…8 pin (run 2) Commits the tb21 Meli-solo condition that until now lived only as untracked files on the runner box, plus the take-2 variant Tyler asked for (thread 74a65a09, event f7457e0b): - tb-meli-solo.yaml + personas/bench/meli-solo.md: the exact manifest and persona (sha256 de633019...) that produced run tb21-solo-1 on the gmicloud/fp8 pin. Recorded for provenance. - tb-meli-solo-baseten.yaml: identical cell, condition tb-meli-solo-baseten, priced at the baseten/fp8 listed rates (prompt $0.13/M, completion $0.26/M — OpenRouter endpoints API, 2026-08-07). - testbed/endpoints/openrouter-baseten.json: the provider pin, OPENROUTER_PROVIDER_ORDER=baseten/fp8. A separate endpoint file so openrouter-live.json (which run 1's provenance points at) stays byte-identical. Run 2 executes on the post-merge harness (9cafe8f, PR #5145), which carries #4896 (vision-input recovery) and #5136 (explicit-mention task post) that run 1 predated. Originating Buzz thread: buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=74a65a0990fd2197882b66b5ea2707169d4a3dbd2020d1610c45150fb99f140b Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
2 participants
Problem
buzz-dev-mcpadvertisesview_imageto every agent regardless of whether the session's model accepts images. When a text-only model (e.g. DeepSeek V4 Flash) takes the bait, the image lands in session history and every subsequent LLM request 404s withNo endpoints found that support image input. The error was classified asLlmModelNotFoundand propagated fatally out of the turn loop — history stays poisoned, buzz-acp retries the batch with exponential backoff, and the session burns its entire clock doing no work. In a recent trial run, all 57 trials that calledview_imageon a text-only model died this way; none recovered.Fix
Capability-gating the advertised tool isn't reliable — there is no image-capability metadata at the agent layer across providers. Instead, recover at the turn loop:
AgentError::UnsupportedImageInput, classified narrowly on the exact provider phraseNo endpoints found that support image inputon both the generic 404 path and OpenRouter's 404 path. Unknown-model 404s and OpenRouter parameter-routing 404s keep their existing classifications. No deterministic retry.RunCtx::runstrips every image block from history — keeping the tool result (and therefore tool-call/result pairing) intact — marks the resultis_error, appends actionable model-facing guidance ("The current model does not support image input. The image was removed from conversation history so this turn can continue. Use a text-based inspection tool…"), and continues the same turn. Base64 never replays again.Tests
fake_llm.rs+fake_mcp.rs): tool call → MCP image result → 404 unsupported-image → same-turn recovery. Captured requests prove round 2 carried the image, round 3 replays no image, carries the guidance text, preserves pairing, and endsend_turn.removed == 0guard survived the suite, andmax_roundsdefaults to unlimited in production, so this branch needed direct coverage.Verified at
a210305019b33d5f56677b4c82bab79e4ac52d24:cargo test -p buzz-agent(full package, 381 unit + all integration suites) green;clippy --all-targets -D warningsgreen;fmt --checkgreen; pre-push hooks (rust-tests, desktop-tauri-checks, branch-skew) green.Scope of the classification guarantee: the classifier runs in the shared
post()(which Anthropic and OpenAI paths route through) and inopenrouter_post()— i.e., every 404 path inllm.rs. It only runs on 404 responses; providers that reject images with a different status (e.g. a 400) are out of scope for this PR — see the review-comment discussion for why broadening the phrase list alone would not cover them.Authored by Wren, loop-guard test by Sami, reviewed by Eva.